Overview

  • What is reproducible research?
  • Why do we care?
  • Why reproducibility questions arise?
  • The cost of reproducibility
  • Reproducibility and statistics
  • Current status of reproducibility
  • What can we do?

What is reproducible research?

Reproducibility and scientific progress

  • Science is the systematic enterprise of gathering knowledge about the universe and organizing and condensing that knowledge into testable laws and theories
  • The success and credibility of science are anchored in the willingness of scientists to expose their ideas and results to independent testing and replication by other scientists.

What is reproducible research?

  • Reproducibility
  • Replicability
  • Repeatability
  • Reliability
  • Robustness
  • Generalizability

What is reproducible research?

  • Transparency
  • Open Science
  • TRUTH

What is reproducible research?

Reproducible research is the ultimate standard for strengthening scientific evidence by independent:

  • Investigators
  • Data
  • Analytical methods
  • Laboratories
  • Instruments

Why do we care?

More data = more chance for errors

  • High-throughput biology generates volumes of data
  • Data-generating technologies are increasingly used to make clinical recommendations and treatment decisions
  • A problem may be overlooked .. Published .. Get in clinical trials

More data = more chance for errors

Poor medical tests getting to patients

Clinical trials based on flawed and fraudulent data

Clinical trials based on flawed and fraudulent data

- Described drug response “gene signatures” in NCI60 cell lines

- Demonstrated these “signatures” correspond to patient-specific signatures and can be used to predict patient response to the drugs

Biostatisticians spot errors

“Off-by-one” error

Published                          Replicated

...
[3,] 1881_at       1882_g_at
[4,] 31321_at      31322_at
[5,] 31725_s_at    31726_at
[6,] 32307_r_at    32308_r_at
...

New signatures continue to be published

Initial data

    Sample   Response GSM44303 RES GSM44304 RES GSM9653  RES GSM9653  RES GSM9654  RES GSM9655  RES GSM9656  RES GSM9657  RES GSM9658  SEN GSM9658  SEN RES/SEN - resistant/sensitive
     

Initial data

    Sample   Response GSM44303 RES GSM44304 RES GSM9653  RES GSM9653  RES GSM9654  RES GSM9655  RES GSM9656  RES GSM9657  RES GSM9658  SEN GSM9658  SEN RES/SEN - resistant/sensitive
     

More data added

    Sample   Response GSM44303 RES GSM44304 RES GSM9653  RES GSM9653  RES GSM9654  RES GSM9655  RES GSM9656  RES GSM9657  RES GSM9658  SEN GSM9658  SEN RES/SEN - resistant/sensitive
    GSM9694 RES GSM9695 RES GSM9696 RES GSM9698 RES GSM9699 SEN GSM9701 RES GSM9708 RES GSM9708 SEN GSM9709 RES GSM9711 RES

More data added

    Sample   Response GSM44303 RES GSM44304 RES GSM9653  RES GSM9653  RES GSM9654  RES GSM9655  RES GSM9656  RES GSM9657  RES GSM9658  SEN GSM9658  SEN RES/SEN - resistant/sensitive
    GSM9694 RES GSM9695 RES GSM9696 RES GSM9698 RES GSM9699 SEN GSM9701 RES GSM9708 RES GSM9708 SEN GSM9709 RES GSM9711 RES

Summary of the Duke case

  • A total of 162 co-authors
  • 40 papers
  • Two-thirds are partially or completely retracted

IOM guidelines on translational omics

PubMed stats on “Reproducible research” vs. “Retraction”

The cost of reproducibility

Irreproducfibility ranges from 51% to 89%

Cost of irreproducibility

Why reproducibility questions arise?

Patterns in the noise

  • Humans are good at recognizing patterns

Human beings do not have very many natural defenses. We are not all that fast, and we are not all that strong. We do not have claws or fangs or body armor. We cannot spit venom. We cannot camouflage ourselves. And we cannot fly. Instead, we survive by means of our wits. Our minds are quick. We are wired to detect patterns and respond to opportunities and threats without much hesitation.

  • Nate Silver

Patterns in the noise

Patterns in the noise

Irreproducibility in high-throughput biology

  • Our intuition about patterns in high dimensional data quickly drops with the increased dimensionality of the data

  • We rely on computation to uncover patterns

  • P values, the ‘gold standard’ of statistical validity, are not as reliable as many researchers assume.

The chance of being wrong

The chance of being wrong

The chance of being wrong

  1. In evaluating any study try to take into account the amount of background noise. That is, remember that the more hypotheses which are tested and the less selection which goes into choosing hypotheses the more likely it is that you are looking at noise.
  2. Bigger samples are better. (But note that even big samples won't help to solve the problems of observational studies which is a whole other problem).
  3. Small effects are to be distrusted.
  4. Multiple sources and types of evidence are desirable.
  5. Evaluate literatures not individual papers.
  6. Trust empirical papers which test other people's theories more than empirical papers which test the author's theory.
  7. As an editor or referee, don't reject papers that fail to reject the null.

 

  • John Ioannidis “Why Most Published Research Findings Are False” PLOS Medicine 2005

http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Understanding the p-value

Understanding the p-value

  1. P-values can indicate how incompatible the data are with a specified statistical model.
  2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
  3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
  4. Proper inference requires full reporting and transparency.
  5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
  6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

P-value warning: consult a statistician before the experiment

Current status of reproducibility

Focus on preclinical research

Focus on preclinical research

NIH focus on openness

NIH focus on openness

NSF stance on openness

NSF stance on openness

Reproducibility initiatives

Reproducibility initiatives

Reproducibility initiatives

Reproducibility guidelines

  • ARRIVE – Animal Research Reporting of In Vivo Experiments
  • CONSORT – Consolidated Standards of Reporting Trials
  • SPIRIT – Standard Protocol Items: Recommendations for Interventional Trials
  • STROBE – The Strengthening the Reporting of Observational Studies in Epidemiology
  • (STARD) TRIPOD – Transparent Reporting of a multivariable prediction model for Individual PROgnosis of Diagnosis
  • REMARK – REporting recommendations for tumour MARKer prognostic studies

http://www.equator-network.org/ - over 300 reporting guidelines